Multifabric global header

ABSTRACT

A Fibre Channel router used to join fabrics. EX_ports are used to connect to the fabrics. The EX_port joins the fabric but the router will not merge into the fabric. Ports in the Fibre Channel router can be in a fabric, but other ports can be connected to other fabrics. Fibre Channel routers can be interconnected using a backbone fabric. Global, interfabric and encapsulation headers are developed to allow routing by conventional Fibre Channel switch devices in the backbone fabric and simplify Fibre Channel router routing. Phantom domains and devices must be developed for each of the fabrics being interconnected. Front phantom domains are present at each port directly connected to a fabric. Each of these is then connected to at least one translate phantom domain. Zoning is accomplished by use of a special LSAN zoning naming convention. This allows each administrator to independently define devices are accessible.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is also related to U.S. patent applications Ser. Nos.______, entitled “Multifabric Zone Device Import and Export,” by DanielChung and Dennis Makishima; ______, entitled “Multifabric CommunicationUsing a Backbone Fabric,” by Dennis Makishima and Daniel Chung; ______,entitled “Interfabric Routing Header for Use with a Backbone Fabric,” byRobert Snively, Steve Wilson, Ed McClanahan, Dennis Makishima and DanielChung; and ______, entitled “System and Method for Providing Proxy andTranslation Domains in a Fibre Channel Router,” by Dennis Makishima andDaniel Chung, all filed concurrently herewith and hereby incorporated byreference.

This application is also related to U.S. patent application Ser. No.10/356,392, entitled “Method and Apparatus for Routing Between FibreChannel Fabrics,” by Chris Del Signore, Vineet Abraham, SathishGnanasekaran, Pranab Patnaik, Vincent W. Guan, and Balakumar N. Kaushik,filed on Jan. 31, 2003, which is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention generally relates to storage area networking, and moreparticularly to interconnecting multiple fabrics in a storage areanetwork.

2. Description of the Related Art

Storage area networks (SANs) are used to allow multiple hosts to connectto multiple storage devices in most applications. This allows sharing ofexpensive storage resources. Typically storage area networks aredeveloped using Fibre Channel switches. Often a storage area network isused if a particular application needs the SAN capabilities, so a smallisolated SAN is developed. In many cases, a different department orapplication will need a SAN and will have one developed for it as well.As this progresses, multiple isolated SANs are developed. However, itwould be desirable to have connectivity between these isolated SAN toprovide overall management and even better use of the various resourcesbut still maintain the fabrics as separate.

In an alternate scenario, a very large Fibre Channel fabric hasdeveloped. As a Fibre Channel fabric grows in size, its stability may bereduced when an event occurs. In addition, if the fabric is formed ofvarious types of devices, it is also significantly harder to managethese devices so that the fabric is maintained and does not fragment.Therefore, in a large fabric case it would be desirable in manyinstances to split the large fabric into multiple fabrics, each asmaller size, but still allow connectivity.

However, connecting fabrics in a Fibre Channel environment is notreadily performed. If a conventional switch were to be connected, itwould just extend the fabric, which does not accomplish the goal ofconnecting fabrics. Therefore it is desirable to have a device which canconnect multiple fabrics without the fabrics being required to merge.This device preferably allows full speed operation between particularfabrics and can be used on simple or large environments.

SUMMARY OF THE INVENTION

A Fibre Channel router according to the present invention can be used tojoin isolated Fibre Channel fabrics to allow greater interconnection ofdevices and overall improved manageability of the entire network. In afirst embodiment the Fibre Channel router contains EX_ports to connectto each of the existing fabrics. The EX_port is a new port developed tohave the port join the fabric but not to have the router itself mergeinto the fabric. Thus the Fibre Channel router can have ports that aremembers of multiple fabrics. The Fibre Channel router will translateaddresses between the particular fabrics so that devices in thedifferent fabrics can communicate.

In a second embodiment ports in the Fibre Channel router can be includedin a fabric as in a conventional switch, but other ports can beconnected to other fabrics. In this embodiment some of the ports of theFibre Channel router act as conventional switch ports, whereas otherports act as interconnection or EX_ports.

In a third variation a plurality of the Fibre Channel routers accordingto the present invention can be interconnected using a backbone fabric.In this case the Fibre Channel routers have conventional E_portconnections through the backbone fabric to other Fibre Channel routers.The connections of the Fibre Channel routers to the fabrics of interestare done with EX_ports as before. To aid in a routing frames through thebackbone fabric, interfabric headers and encapsulation headers aredeveloped to allow routing by conventional Fibre Channel switch devicesin the backbone fabric.

One complexity arises because phantom or proxy domains and devices mustbe developed for each of the fabrics being interconnected, so that thedevices on a particular fabric are not aware of the various translationswhich must occur. To this end, each Fibre Channel router has at leasttwo domains that it develops. A first set of domains is the frontphantom domains. These are domains present at each port directlyconnected to a fabric. Each of these front phantom domains is thenconnected to at least one translate phantom domain, there being onetranslate phantom domain for each particular fabric to which devicesbeing translated are connected. The use of the front phantom domains foreach connected port and a translate phantom domain for each connectedfabric simplifies overall routing operations and provides redundancy andsimplification of the software required.

In a final embodiment, two fabrics can be joined by having a FibreChannel router connected to each fabric and the Fibre Channel routersinterconnected. In this embodiment the Fibre Channel routers thenperform the necessary isolation between the particular connectedfabrics.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a series of isolated fabrics interconnectedby a Fibre Channel router according to the present invention.

FIG. 2 is a more detailed block diagram of isolated fabrics beinginterconnected by a Fibre Channel router according to the presentinvention.

FIG. 3 illustrates the phantom or proxy devices developed by a FibreChannel router according to the present convention when interconnectingtwo fabrics.

FIG. 4 is a block diagram of a Fibre Channel router according to thepresent invention interconnecting a series of fabrics and acting as aFibre Channel switch in a fabric.

FIG. 5 is a block diagram of a series of Fibre Channel routers accordingto the present invention interconnecting a series of isolated fabricsthrough the use of a backbone fabric.

FIG. 6 is a more detailed view of the interconnections of the FIG. 5.

FIG. 7 is a block diagram of a series of Fibre Channel routers accordingto the present invention interconnecting a series of fabrics using twobackbone fabrics.

FIG. 8 is a block diagram of a series of backbone fabrics joined by abackbone fabric.

FIG. 9 is block diagram of a plurality of Fibre Channel routersaccording to the present invention interconnecting a series of fabricsthrough a backbone fabric with exemplary front and translate phantomdomains utilized to provide the phantom or proxy operations.

FIG. 9A is a more detailed version of FIG. 9 showing all of the frontand translate phantom domains.

FIG. 10 is an illustration of the connections and port numberings forfront and translates phantom domains according to the present invention.

FIG. 11 is a more complicated illustration of the front and translatephantom domains for a plurality of connections.

FIG. 12 is a block diagram of various components of a Fibre Channelrouter according to the present invention.

FIG. 13 is a block diagram of the preferred embodiment of a portprocessor of FIG. 12.

FIG. 14 is block diagram of the various software modules executing onthe port processors and central processor of FIG. 12.

FIGS. 15 is a flowchart of the operation steps of a port processor ofFIG. 12 operating as an EX_port.

FIG. 16 is a flow chart illustration of a port processor of FIG. 12operating as an NR_port.

FIG. 17 is an alternate embodiment illustrating the interconnection ofFibre Channel routers according to the present invention where the FibreChannel routers are connected to the fabrics and are connected to eachother.

FIG. 18 is the embodiment of FIG. 17 with multiple fabrics beinginterconnected.

FIG. 19 is an illustration of a plurality of Fibre Channel routers ofFIG. 17 interconnection a plurality of fabrics.

FIG. 20 is an illustration of multiple fabric zoning according to thepresent invention.

FIG. 21 is a depiction of a global header used in multi-fabric zoningaccording to the present invention.

FIG. 22 illustrates a Fibre Channel frame including various inter-fabricrouting headers according to the present invention.

FIG. 23 illustrates one proposed layout of the fabric routing headeraccording to the present invention.

FIG. 24 illustrates an encapsulation header according to the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to FIG. 1, a Fibre Channel router 100 according to thepresent invention connected to a series of fabrics A-E 102A-102E by aseries of links where a switch in each of the fabrics A-E 102A-102E hasan E_port connected to an EX_port of the Fibre Channel router 100. AnEX_port is, in great respects, an E_port but it does not cause themerger of the remainder of the Fibre Channel router 100 with theconnected edge fabric 102. A host 104 is connected to fabric A 102 A,while a host 106 is connected to a fabric B 102 B. Similarly, storagedevices 108, 110 and 112 are connected to fabrics C-E 102C-102E.

Referring then to FIG. 2, it can be seen that the fabric A 102A includesa series of Fibre Channel switches 120 which are interconnected toconnect to the host 104 and have multiple connections to the FibreChannel router 100. In the second fabric C 102C, the storage device 108is connected to a single Fibre Channel switch 120, which has a singlelink to the Fibre Channel router 100.

In FIG. 3 the Fibre Channel router 100 is connected to the fabric A 102Aand the fabric C 102C. Host 104 is connected to fabric A 102A but due tothe operation of the Fibre Channel router 100 appears as host 104′connected to fabric C 102C. The storage device 108 connected to fabric C102C is presented by the Fibre Channel router 100 on fabric A 102A. Thushost 104 would address storage device 108′, and storage device 108 wouldaddress host 104′. The Fibre Channel router 100 would handle translatingaddresses between the particular devices as necessary, as will bedescribed in more detail below.

In an alternate embodiment illustrated in FIG. 4, the Fibre Channelrouter 100 can be connected to a series of fabrics A, B and C 102 A. TheFibre Channel router 100 also acts as a conventional Fibre Channelswitch in a fabric D 102D. In this case the Fibre Channel router 100connects to another Fibre Channel switch 120, which in turn could beconnected to a storage device 126. Similarly, a host 122 could bedirectly connected to the Fibre Channel router 100 through an F_port anda storage device 124 could be connected through an FL_port. Thus in thepreferred embodiments the Fibre Channel router 100 is a flexible devicewhich can both interconnect fabrics and act as a conventional switch.

Referring now to FIG. 5, a much more complicated interconnection schemeis shown. While the interconnection schemes of the previous figures aresatisfactory for a limited number of fabrics, if a great number offabrics need to be interconnected a plurality of Fibre Channel routers100 must be used. In the illustrated case a plurality of Fibre Channelrouters A-D 100A-100D are used to interconnect a series of fabrics A-H102A-102H. Fibre Channel routers A-D 100A-100D are connected to abackbone fabric 150 using E_ports. The backbone fabric 150 can be formedusing conventional Fibre Channel switches in the preferred embodiment.Thus if host 104 wishes to send a frame to storage device 132, thepacket will travel through fabric A 102 A through Fibre Channel router A100A through backbone fabric 150 to Fibre Channel router C 100C tofabric H 102H to storage device 132. To transfer frames through thebackbone fabric, 150 the Fibre Channel routers A-D 100A-100D add severalheaders to the frames. At the EX_port, in the preferred embodiment, theFibre Channel router 100 adds a global header, an encapsulation header,and a fabric routing header. These headers are described below in moredetail. Then in a receiving Fibre Channel router C 100C, in the aboveexample, a port, referred to as an NR_port, will remove the global,encapsulation, and fabric routing headers and develop a normal packet orframe which is sent to fabric H 102H.

FIG. 6 is an alternate embodiment in which a host 160 and a storagedevice 162 are also connected to the backbone fabric 150. FIG. 6additionally illustrates connections to specific ports on the FibreChannel routers A-C 100A-100C. In the preferred embodiment, each port ina Fibre Channel router 100 is actually a dual-ported device, one portexternally facing and one port internally facing, with the preferredFibre Channel router 100 treating the internal port connections as beingconnected by a fabric. As shown in FIG. 6, for ports connected to theedge fabrics 102 the external port is considered to be the EX_port andthe internal port is considered to be the NR_port for conceptualpurposes. For other designs the internal interconnections of the portsmay be different, but it is believed that a similar nomenclature wouldbe useful and applicable.

Connecting an EX_port to an E_port has the following properties. AnEX_port connects to an existing fabric as an E_port. The EX_portimplements enough of a subset of E_port functionality to allow fabricinitialization and to allow critical fabric services to operate to makethe fabric usable. The EX_port-to-E_port link is essentially aninterswitch link (ISL). The EX_port will not merge the fabric that it isjoining with any other Fibre Channel router-connected fabrics (the FibreChannel router 100 may be connected to several fabrics). EX_portspresent proxy devices representing real devices in remote fabrics inorder to facilitate interfabric communication between devices. Incertain embodiments, for simplicity, switching does not occur betweenmultiple EX_ports of a Fibre Channel 100 router that are connected tothe same edge fabric. Multiple EX_ports from the same Fibre Channelrouter 100 or from independent Fibre Channel routers 100 can connect tothe same edge fabric in a redundant/resilient manner. Using methodsdescribed below, the port IDs of the proxy devices are preserved by theFibre Channel routers 100 to provide as seamless of a failover aspossible from the host's perspective. The IDs are preferably alsopreserved across Fibre Channel router 100 reboots and temporary loss ofthe backend connection to the device. The Fibre Channel router 100achieves this functionality by emulating shared phantom domains asdescribed more fully below.

Fibre Channel routers 100 have “router” ports known as NR_ports thatsend and receive frames containing Fibre Channel router “global header”information. In this description, as noted above, the phraseologyEX_port is generally used for the externally facing portion of a portand NR_port is used for the internally or backbone facing portion of aport, but it is understood that they are the same physical port. Thus aframe is received at an EX_port, transmitted through the backbone fabric150 to an NR_port and transmitted to the final fabric. The reverse flowwould follow the same path but the port names would be changed to keepthe context clear. Global header information includes details thatfacilitate route decisions and address translations. Frames traversingbetween networked Fibre Channel routers 100 travel effectively fromNR_port to NR_port. Traffic from many devices from many fabrics can becarried across an intermediate or backbone fabric 150 via a single pairof Fibre Channel routers 100. If an FCIP tunnel exists in the backbonefabric, with Fibre Channel routers 100 on each side of the tunnel, thentraffic for multiple fabrics can traverse across a single FCIP tunnel.For certain embodiments traffic for multiple fabrics can be carried overa single backbone ISL (interswitch link). In other embodiments globalend-devices can insert the global headers into frames and direct framesdirectly to a Fibre Channel router's 100 NR_port. Note that the NR_portconcept still applies to single Fibre Channel router 100 configurationsbecause the backbone fabric is essentially the internal fabric in asingle Fibre Channel router 100 configuration.

Three separate headers are used to carry frames between EX_ports andNR_ports. The encapsulation header is the first header and specifiessource and destination NR_ports in the SID and DID fields. This headerhas an identical format as a normal FC-FS frame header. Therefore legacyswitches in a backbone fabric can route with this header. “GlobalHeader” information is contained in two headers. Some of the globalheader is included in a short header known as the interfabric addressingheader. The remaining “global header” information is contained in thenormal FC-FC frame header that is contained in the frame, is describedabove.

FIG. 7 is similar to FIG. 5 except that two backbone fabrics A and B150A, 150B are utilized. In this case backbone fabric A 150A is used tointerconnect Fibre Channel router A 100A to Fibre Channel router C 100C,while Fibre Channel router B 100B is connected through backbone fabric B150 B to Fibre Channel router D 100D.

FIG. 8 illustrates a larger example. In this example there are threebackbone fabrics A, B and C 150A, 150B and 150C with associated FibreChannel routers 100 and fabrics 102. They are connected through FibreChannel routers J, K and L 100J, 100K and 100L to a backbone ofbackbones fabric 175. In this instance the Fibre Channel routers 100connect with E_Ports to each fabric 150, 175. The Fibre Channel routers100 in this case are used to keep the fabrics from merging and onlyperform routing of the frames, as the proper headers are already inplace in the frames and no conversions need to be done. The headeraddition or removal and needed conversions will be done be the FibreChannel routers 100 attached between the edge fabrics 102 and thebackbone fabrics 150.

FIG. 9 illustrates the use of front phantom domains and translatephantom domains to allow the Fibre Channel routers 100 to perform thenecessary translations and yet allow efficient operations. A frontphantom domain 200 is provided for each EX_port of a Fibre Channelrouter 100 connected to a fabric. In the illustrated embodiment thereare front phantom domains 200A, 200B, and 200C for the connections tofabric A 102A. Each of these front phantom domains 200 is in turnconnected to a translate phantom domain 202 such as 202AB and 202AC. Atranslate phantom domain 202 is provided for each fabric wheretranslations are necessary, such as in the illustrated case fabric B102B and fabric C 102C. These translate phantom domains 202AB and 202ACpresent the phantom or proxy storage devices as devices attached to thetranslate phantom domain 202. There is an owner front phantom domain foreach translate phantom domain. The original owner front phantom domainis the first front phantom domain created which needs to connect to thetranslate phantom domain. In the illustrated embodiment, translatephantom domain A_B 202AB presents storage device 108′ and storage device110′, which correspond to storage device 108 and storage device 110connected to fabric B 102B. Translate phantom domain A_C 202AC presentsstorage device 112′. FIG. 9 shows only the front and translate phantomdomains 200 and 202 necessary for connection to fabric A 102A forsimplicity.

Sharing a translate phantom domain 202 between Fibre Channel routers100, as opposed to providing a translate phantom domain from each FibreChannel router and resultant duplication of proxy devices, allows forload balancing by the edge fabric and is less disruptive should a linkfail as the proxy device ID will not need to be changed. Providing onetranslate phantom domain 202 per fabric, as opposed to a singletranslate phantom domain for all fabrics, similarly provides loadbalancing and link failure advantages.

FIG. 9A shows the full front phantom domain 200 and 202 environment forall of the fabrics 200 in FIG. 9A. The fabrics A, B and C 102A-C areshown in an expanded format to indicate that the front and translatephantom domains are part of the respective fabric. For example, fabric B102B includes front phantom domain D 200D and translate phantom domainsB_A and B_C 200BA and 200BC. Translate phantom domain B_A 200BA presentshost 104′, while translate phantom domain B_C 200BC presents storagedevice 112″, the second proxy for that particular storage device. Thusit can be seen that each fabric includes translate phantom domains foreach of the other fabrics being connected. In the illustrated embodimentthis means each fabric includes two translate phantom domains.

Each front phantom domain is emulated as being connected to each of thetranslate phantom domains in the particular fabric. Thus front phantomdomains E and F 200E, 200F are connected to translate phantom domainsC_A and C_B 202CA, 202CB. Translate phantom domain C_A 200CA presentshost 104″, while translate phantom domain C_B 200CB presents storagedevices 108″ and 110″. Storage device 112 thus addresses host 104″connected to translate phantom domain C_A 200CA.

While the phantom domains and proxy devices described above provide theaddressing and fabric construct needed by the various mode devices, adescription of actual frame flows in such an environment is consideredvery helpful. In FIG. 9A solid line links are physical links, dottedline links are emulated or phantom links and the dot-dash line is anexemplary frame path. Host 104 provides a frame addressed to storagedevice 112′. The frame travels to switch 120A and then to port 204 ofFibre Channel router A 100A. As described more fully below, port 204performs the necessary address translations and develops interfabric andencapsulation headers for the frame. The Fibre Channel router A 100Athen routes the frame to port 206, which is connected to the backbonefabric 150. This routing is based on the routing tables present in theFibre Channel router A 100A. The backbone fabric 150 then routes theframe to port 208 on Fibre Channel router C 100C, the port connected tothe backbone fabric 150. The Fibre Channel router C 100C routes theframe to port 210. Port 210 acts as the NR_port and removes theinterfabric and encapsulation headers and performs the final addresstranslation. Port 210 then provides the frame to switch 120C, whichroutes the frame to storage device 112. It is understood that thesephantom domains are software constructs used by various softwareentities in the Fibre Channel routers 100 to perform the necessarytranslations operations.

FIG. 10 illustrates the case of a front phantom domain, in this casewith a domain ID of 3 which was developed during typical domain IDdevelopment in the fabric. The front phantom domain connects through theport in the Fibre Channel router 100 connected to the edge fabric 102.The front phantom domain is connected to a translate phantom domainrepresenting one of the particular fabrics. In this case it representsfabric 4. The front phantom domain uses port number 0 to connect to theactual port and a port number equal to the translate phantom domain toconnect to the translate phantom domain. The translate phantom domainconnects to a front phantom domain at a port number one less than thefront phantom domain ID. Thus in the example of FIG. 10, the frontphantom domain uses a port 4 to connect to port 2 of the translatephantom domain. The example shown in FIG. 11 has three front phantomdomains having IDs of 3, 5 and 6 connecting to the translate phantomdomain representing fabric 4. Each of the front phantom domains willhave its port 4 connected to the translate phantom domain while thetranslate phantom domain will use ports 2, 4 and 5 to connect it to thevarious front phantom domains.

In the operation of the Fibre Channel routers 100, each of the front andtranslate phantom domains have both World Wide Names (WWN) and domainIDs. It is necessary for phantom domains and phantom domain ports tohave WWNs for operational (i.e. fabric formation, traffic generation,etc) and management purposes. A front phantom domain requires a node WWNas well as a port WWN to establish a neighbor relationship with an edgefabric through Fibre Channel ELP frames. The node WWN is used again whenthe front phantom domain initiates an RDI request (RDI=Request DomainID) to receive a valid domain ID from the Principal Switch in the edgefabric. The ports between the front and translate phantom domains do notrequire port WWNs from an operational perspective since the ELP betweena front phantom domain and the translate phantom domain is implicit.However, the translate phantom domain still requires a node WWN in orderto receive a domain ID from the Principal Switch. With the exception ofthe EX_ports, phantom domain ports do not require WWNs from anoperational perspective. However, many management applications rely uponport WWNs as a key for port objects, which may be needed to build thetopology. As a result, port WWNs need to be generated for the portsconnecting the front and translate domains as well as the translatedomain proxy device ports. Since the Fibre Channel router 100 requiresmany WWNs to be generated, NAA type 2 and type 5 WWN formats are needed.In addition, the Fibre Channel router 100 makes use of the various MACaddresses assigned to the platform to derive WWNs. One base MAC addressis assigned per the Fibre Channel router 100 in the preferredembodiment. In addition, each external port is assigned a MAC addresssince the ports can be Gigabit Ethernet ports in the preferredembodiment. Each port MAC address is derived from a base MAC address bysetting the last nibble to the port number.

The node WWN of a front phantom domain is an NAA type 5 WWN derived fromthe MAC address of the platform and EX_port port number. In the NAA type5 format, 4 bits denote the NAA type (5) and 48 bits are used for theMAC address which contain the serial number and manufacturer's codewhich leaves 12 available bits. The top 4 bits of the 12 available bitsare assigned to 0xE, denoting “front phantom domain”, and last byte isassigned with the EX_port number. In one embodiment up to 16 WWNs may beused for front phantoms since there is a front phantom domain perEX_port and there are 16 ports in the Fibre Channel router 100. In otherembodiments there can be greater or fewer ports. The following describesthe format of the front domain WWN:

5n:nn:nn:nn:nn:nn:nE:xx, where the n's represent the MAC address, Edenotes “front phantom domain” and xx=port number.

The port WWN used for the front phantom domain's external port, which isthe same port as the physical external port, is an NAA type 2 port WWNand is the WWN of the EX_port. This WWN is derived from the MAC addressof the platform and the external port number. Up to 16 WWNs may be usedfor these ports in one embodiment and greater or fewer in otherembodiments. The following describes the format of the front phantomdomain EX_port WWN:

20:xx:nn:nn:nn:nn:nn:nn, where the n's represent the MAC address of Marsand xx=port number.

The port WWNs for the front phantom domain ports that connect to thetranslate domain(s) are NAA type 2 names with identical format as above.These names are derived from the MAC address of the external port (asopposed to the MAC address of Fibre Channel router 100) and the portnumber of the phantom port. There may be up to 128 translate domainsthere can be up to 128 front domain ports in one embodiment. Since thereare up to 16 front domains, there may be up to 16*127 or 2032 frontdomain port WWNs. The following describes the format of the front domainphantom port WWN:

20:xx:nn:nn:nn:nn:nn:nn, where the n's represent the MAC address ofexternal port and xx=phantom port number.

The Fibre Channel router 100 provides unique translate phantom domainWWNs within the SAN in order to satisfy management applications that aredesigned assuming globally unique WWNs. Translate phantom domain nodeWWNs are allocated from a pool of WWNs on a first come first servebasis. They are allocated in a sequential manner until the pool isexhausted. Once the pool is exhausted, allocation resumes from the startof the pool. Since the pool is sufficiently large, WWN conflicts arehighly unlikely. The node WWN used is an NAA type 5 WWN derived from theMAC address of the EX_port of the owner front phantom domain, the firstfront phantom domain created which needs the particular translatephantom domain. As mentioned previously, there are 12 available bits inthe NAA type 5 format. The top 4 bits of the available bits are set to0xF, denoting “translate phantom domain,” and next 8 bits are fullyutilized to construct the pool. This allows creation of a pool of 256*16or 4096 node WWNs per Fibre Channel router 100. Since translate phantomdomain ownership can change, front phantom domains sharing a translatedomain employ an inband protocol to exchange translate phantom domainnode WWNs as well as domain IDs. The following describes the format ofthe translate phantom domain node WWN:

5n:nn:nn:nn:nn:nn:nF:xx, where the n's represent the MAC address ofEX_port of the original owner front phantom domain, F denotes “translatephantom domain” and xx=8 bits used to generate as many WWNs as possible.

Translate phantom domains have ports that connect to front phantomdomains and ports that connect to proxy devices (phantom F_ports). Thereare more translate phantom domain theoretical ports than available WWNssince there can be up to 127 translate domains per fabric each with 256ports in certain embodiments. Therefore, translate phantom domain portWWNs are allocated from a pool of WWNs on a first come first servebasis. They are allocated in a sequential manner until the pool isexhausted. Once the pool is exhausted, allocation resumes from the startof the pool. Since the pool is sufficiently large, WWN conflicts arehighly unlikely. These WWNs are NAA type 5 format with the first nibbleof the available 12 bits as 0xA-0xD. The WWNs use all of the port MACaddresses for the MAC address portion of the WWN and utilize allremaining 8 bits. This allows creation of a pool of 256*16*4 or 16,384WWNs. This number of WWNs is ample for scalability. For example, if aFibre Channel router 100 has 15 front phantom domains, each connected toa different fabric and 127 translate phantom domains per fabric, thisresults in 15*127 (1905) translate phantom domains. Assuming eachtranslate phantom domain connects to 4 front phantom domains thisresults in 7,620 port WWNs. If 1,000 proxy devices are created in eachdomain fabric, this is still safely within the limits of the port WWNpool. The owner front phantom domain is responsible for generating thetranslate phantom domain port WWNs. Since translate phantom domains areshared among multiple front phantom domains, front phantom domainsemploy an Inband protocol to exchange translate phantom domain portWWNs. The following describes the format of the translate phantom domainport WWN:

5n:nn:nn:nn:nn:nn:n[A-D]:xx, where the n's represent the MAC address ofall EX_ports of the platform, A-D denotes “translate phantom domainports” and xx=8 bits used to generate as many WWNs as possible.

As mentioned previously, translate phantom domains may be shared amongseveral front phantom domains that are presented by multiple physicalFibre Channel routers 100. The Fibre Channel router 100 with the ownerfront phantom domain allocates the node and port WWNs to the translatephantom domain. If the owner front phantom domain becomes unavailable,translate phantom domain ownership can change and may possibly change toa different physical Fibre Channel router 100. This potentially depletesthe finite number of translate phantom domain node and port WWNs fromthe original Fibre Channel router 100 pool. When a WWN pool is nearlyexhausted, a warning event is generated. When the pool is exhausted,allocation resumes from the start of the pool. Another event isgenerated warning the user that the pool has been reset. Once the poolhas been reset it is possible for WWN duplication to occur. Since theWWN pools are sufficiently large, there is a low probability ofexhausting the WWN pool. In addition, the rate of depletion is expectedto be flat once the SAN is established. Even in the event that a frontphantom domain loses ownership of a translate phantom domain, it islikely that it will come back online in the same edge fabric. Thetranslate phantom domain it originally owned will likely still exist inthe fabric, which would not cause WWN depletion. If the front phantomdomain comes back online in the same fabric and is required to establisha new translate phantom domain for the same remote fabric, then new WWNswill be used from the top of the pool. This does cause WWN depletion,however, in this case the original translate phantom domain no longerexists so the WWNs originally used for the translate phantom domain aresafe to reuse in the event the pool needs to be reset. Therefore, nospecial action needs to be taken when pool reset occurs. To guaranteeelimination of WWN reuse causing management application issues, the usercan disable all EX_ports attached to fabrics where high WWN depletionoccurred (e.g. fabrics with a large number of translate phantom domainports where EX_ports went offline/online frequently).

It is understood that the phantom domains are software constructs in thepreferred embodiment, though they could be developed in hardware. TheFibre Channel router 100 handles both the fabric level operations ofeach of these domains, such as communications with the principal switchto receive a domain ID; respond to ILS frames; perform FSPFdeterminations and so on, and frame level operations such as addresstranslation and routing, thought hardware assist is preferred in theframe level operations.

EX_ports support a subset of fabric services. See list below:

Zoning—Zoning ILSs are supported in order to satisfy other switcheswithin the fabric that want to share the zoning database. In addition,the Fibre Channel router 100 searches the zoning database to extract theWWNs for interfabric device sharing and to potentially determine thezoning policy for the imported/exported devices.

Fabric Build—Enough features are supported to convince the fabric of thepresence of phantom switches that become the phantom domains. EX_portsprovide fabric parameters (such as E_D_TOV, R_A_TOV and PID format) atport initialization time. Users are allowed to configure theseparameters to ensure EX_ports are allowed to merge with the edgefabrics. Alternatively, these parameters can be detected from the edgefabric and set to match. Note that the TOV values are “fake” values usedonly to facilitate port initialization.

FSPF—Enough features are supported to convince the fabric of the phantomswitch topology presented by the Fibre Channel router 100. Routes to thetranslate phantom domains are advertised as maximum cost links toprevent the fabric from establishing intrafabric routes using theselinks. Should the loss of a path cause the fabric to want to utilizethese links for intrafabric routing, the Fibre Channel router 100 willdisable the EX_ports in some embodiments.

NS/RSCN—A small subset of NS (Name Server) features are supported inorder for the Fibre Channel router 100 to satisfy other switches withinthe fabric that want to share NS databases. In addition, the FibreChannel router 100 needs to determine the presence and state of devicesto be exported. NS queries from end devices are not supported in certainembodiments for simplicity.

Management Server (MS)—A small subset of features are “faked” so thatthe MS platform support does not need to be disabled within an edgefabric. In addition, a small subset of the topology commands supported afabric architecture so that the API can create its object hierarchy(topology). From the backbone fabric 150 perspective, the NR_ports andEX_ports are essentially “invisible” as the E_port MS implementationdoes not return port information for “special” ports (i.e. EX_ports,iSCSI ports, etc).

The management model of the Fibre Channel router 100 is that of a“switch with special ports.” Port management involves port configurationand status.

Port configuration includes the following:

EX_port mode Enable/Disable—Ability to enable/disable EX_portfunctionality per port and the ability to select the fabric ID for aport.

Fabric ID—Selecting the fabric ID for a port provides the mechanism toidentify fabrics. For example, the way a fabric becomes “fabric 1” is byconfiguring a EX_port with a fabric ID of “1”.

Fabric Parameters—Fabric parameters are exchanged by E_ports at portinitialization. Incompatible fabric parameters cause E_ports to segment.EX_ports exchange these parameters with the neighbor E_port. Theseparameters are typically administered at a switch level. Since EX_portsrepresent a front phantom domain, these parameters need to be managed ona port by port basis. The fabric parameters include:

R_A_TOV

E_D_TOV

PID format

Front Phantom Domain ID—When configured by a user, the front phantomrequests this domain ID from the Principal Switch. If a different domainID is issued, the new value is accepted and persistently stored as thenew preferred domain ID.

Another aspect of port management is port status, which includes statuson the above listed configurable parameters. Additional parametersinclude last error (e.g. FID Conflict, FID oversubscribe, etc), switchnode WWN of the front phantom domain for this EX_port and information onthe edge fabric such as Principal Switch information (e.g. WWN, domainID, etc.).

When an EX_port receives a EFP or Enable fabric port frame, from thatpoint on, fabric ILSs are handled by the Fibre Channel router 100 in apassive manner. Passive manner means is that each ILS sent by aneighboring or a principal switch in an edge fabric drives a fabricstate machine for front and translate phantom domains.

The front phantom domain is required to handle all the ILSs listedbelow. The front phantom domain also handles any RDI responses on behalfof translate phantom domains that it owns, and INQ requests fortranslate phantom domains it may or may not own. An RDI response ishandled exclusively by owner front phantom domains because owner frontphantom domains initiate RDI requests on behalf of translate phantomdomains. Either owner or non-owner front phantom domains can respond toINQ requests on behalf of translate phantom domains because INQ responseinformation is calculated based on a fixed algorithm.

An EX_port goes through 4 fabric states: Init, Merge, Rebuild, and Done.When an EX_port comes online, it goes to Init state. From there, theEX_port can go to either Merge or Rebuild. Merge is the case wherephantom switches are “merging” to edge fabrics after the edge fabricshave completed Principal Switch selection. The Merge path is taken onlyif the first EFP is non-empty once the EX_port has come online. Rebuildis the case where phantom switches are “rebuilding” with edge fabricssince Principal Switch selection has not taken place. The rebuild pathis taken if empty EFPs are received once the EX_port has come online ora BF request has been received. Once the EFP payload is received withthe EX_port's front phantom domain ID embedded, the EX_port goes toDone.

EX_ports send out INQs into the connected edge fabric to find realswitch information in the edge fabric. Outbound INQ ILSs are triggeredwhen an INQ ILS is received for an EX_port front phantom domain.

The following fabric ILSs and states are handled.

EFP

When receiving an empty EFP as the first EFP after the port going onlineor right after receiving a BF, the front phantom domain responds byaccepting the EFP and sets the port state to Rebuild.

When receiving a non-empty EFP right after the port receiving the firstEFP after goes online, the front phantom domain responds by acceptingthe EFP. The edge fabric Principal Switch WWN is remembered and an RDIrequest is sent to receive a valid domain ID for the front phantomdomain. Anytime a non-empty EFP is received with the front phantom'sassigned domain ID, the following algorithm is executed.

Port state is set to done.

Update the NR_port database to get an updated list of remote edgefabrics that are reachable.

Go through the domain list and add or remove domains from the lastremembered list. In certain embodiments the Domain ID and switch WWNcombination is remembered.

If any phantom domain has been removed, send out a full LSR database tothe edge fabric to update routing.

All received EFPs are accepted as is and the EX_port generates no EFP onits own.

DIA

A DIA request is always accepted. If the port state is in Rebuild, anRDI request is sent to receive a valid domain ID for the front phantomdomain.

BF

When receiving a BF request at anytime, it is accepted. The Port stateis updated to Rebuild. All other port states, except domain IDs for allphantom domains, are remembered until overwritten by the fabricreconfiguration process.

RDI

When sending an RDI request for front or translate phantom domains,previously saved domain IDs are used if available. Otherwise, a defaultdomain ID of 1 is requested. When receiving an RDI response targeted fora front phantom domain, the domain ID assigned is remembered and savedin non-volatile memory. In addition, if the front phantom domain ownsany translate phantom domain and valid domain IDs are assigned to thetranslate phantom domains, RDIs are sent by the front phantom domain onbehalf of the translate phantom domain. When receiving an RDI responsetargeted for a translate phantom domain, the domain ID assigned andswitch WWN are remembered and saved in non-volatile memory. If thetranslate phantom domain was ever allocated before, a full LSR is sentto update the addition of the translate phantom domain. Otherwise,domain only LSR is sent to initiate a full LSR sequence with theneighboring switch. If the newly assigned domain ID is different thanthe previously assigned domain ID, imported devices are re-importedusing the proxy Port ID recalculated based on the new domain ID.

INQ

When receiving an INQ request, all the response payload fields are fixedusing an algorithm. Switch names are fixed at “ff_<domain>_<port number>for front phantom domains and “xf_<domain>_<fabric number> for translatephantom domains. IP addresses are zeroed out. Fake model number andfirmware version are returned to mimic a selected version. The edgefabric switches initiate different NS requests based on the returnedfirmware version number.

Although FSFP standards requires all switches within a fabric tomaintain full path configuration (Link State Database), phantom domainsdo not need full topology information stored to communicate with realswitches in an edge fabric and to fake the real switches in the edgefabric into believing that they have full topology information ofphantom domains. However, more complete or even full topology orsubstitutes can be stored and used if desired. Each front phantom domaingenerates an LSU payload that contains itself and its links to the edgefabric and translate phantom domains. Only when a front phantom domainowns a translate phantom domain, does the front phantom domain need toappend to an LSU payload the translate phantom domain and all links fromthe translate phantom domain to all front phantom domains with access tothe remote edge fabric that the translate phantom domain represents.

All FSPF ILSs are targeted and handled by front phantom domains.Although. FSPF ILSs contain translate phantom domain specific payloadsthat are handled by owner front phantom domains, no FSFP ILSs aretargeted to translate phantom domains.

The following ILSs and states are handled:

HLO

Once a front phantom domain has been assigned a domain ID and asubsequent EFP has been received with a full domain list, a HLO isresponded by another HLO with remote switch ID and local switch ID ofthe original HLO swapped. In other words, HLO's from an EX_port aregenerated only when a HLO from neighbor is received.

Once a full LSU has been sent, if a neighboring switch sends a one-wayHLO, which contains 0xffffffff as the remote switch ID, the LSU state isre-initialized to the beginning and the domain waits for a full LSRupdate from the neighboring switch.

LSU

Once a front phantom domain has been assigned a domain ID and asubsequent EFP has been received with a full domain list, an LSU ishandled in the following manner: A LSA is sent right away. The LSUpayload is examined to remember an incarnation number for valid domainswithin the edge fabric. Otherwise, an initial incarnation number of0x80000001 is used for phantom domains. If the LSU is sent by a realswitch in an edge fabric and an End-of-database has been received, anLSU with front phantom domain ID with no links is sent and subsequentlyanother LSU with end-of-database is send.

LSA

When receiving an LSA for a front phantom domain and the front phantomdomain has just sent its domain only LSU, send the front phantomdomain's full LSU that contains itself, any links originating from thefront phantom domain and any translate phantom domains the front phantomdomain owns and any links originating from the translate phantomdomains.

When receiving an LSA for a translate phantom domain and owner frontphantom domain has already sent a domain only LSU on behalf of thetranslate phantom domain, send the front phantom domain's full LSU thatcontains all translate phantom domain links.

Multiple EX_ports can be used to support load distribution. The EX_portscan be from a single Fibre Channel router 100 or different Fibre Channelrouters 100 within a backbone. The EX_ports can also be from multiplebackbone fabrics 150. The EX_ports create front phantom domains torepresent the ports themselves and create a translate phantom domain foreach remote edge fabric that the local edge fabric communicates to. Asnoted, a reason for creating the front and translate phantom domains areto allow existing FSPF within the local edge fabric to handle loaddistribution, path selection, failure handling, etc. and to preservePIDs assigned to proxy devices. In order for multiple front phantomdomains to create a single translate phantom domain for a particularremote edge fabric, EX_ports must:

Discover all front phantom domains that connect to the same edge fabricand have access to the same remote edge fabric.

Vote one of the front phantom domains to own the remote edge fabric todrive translate phantom domain handling.

When a front phantom domain represented by an EX_port has merged to anedge fabric (the front phantom domain has been assigned a unique domainID by the edge fabric's Principal Switch) FCR_EF_ALIVE messages are sentto all other front phantom domains. An FCR_EF_ALIVE message contains thefront phantom domain ID, the Principal switch WWN of the edge fabric theEX_port is connected to, the switch WWN of Fibre Channel router 100, anda list of edge fabric entries. An edge fabric entry contains, for alledge fabrics minus the one the originating EX_port is attached to, thefabric tag of the edge fabric, the translate phantom domain ID assigned(if not yet assigned, the domain ID is set to 0 and the allocatedtranslate phantom domain node WWN and port WWNs are zeroed out), thelowest front phantom domain ID received so far (initial lowest frontphantom sent out is the local front phantom domain ID), the owner frontphantom domain ID (initial owner front phantom domain ID sent out is 0),and the list of edge fabric switches that have completed handshakes withthe translate phantom domain.

If a front phantom domain receives an FCR_EF_ALIVE message from anotherfront phantom domain connected to the same edge fabric, the list ofreachable edge fabric entries is processed one by one using thefollowing algorithm:

If the front phantom domain receiving the FCR_EF_ALIVE message is notable to reach the fabric specified by the fabric tag in the entry,ignore the entry. Otherwise, continue.

If the front phantom domain owns a fabric that the front phantom domainis no longer able to reach and another front phantom domain has beenelected as an owner, the local front phantom domain is set to non-ownerand cached information is cleared.

If cached information of the remote front phantom domain, whichoriginated the FCR_EF_ALIVE message, previously had an invalid translatephantom domain ID but the FCR_EF_ALIVE message now contains a validtranslate phantom domain ID, a full FSPF is sent to create an uplinkfrom the translate phantom domain to the originating front phantomdomain.

Cache the lowest front phantom domain ID and the owner front phantomdomain ID using the originating phantom domain ID and fabric tag withinthe entry.

If the receiving front phantom domain's owner for the fabric specifiedin the entry has not been set:

If owner front phantom domain ID embedded in the entry is a valid domain(1-239), it is accepted. The receiving front phantom domain's ownerfront phantom domain ID is set to the embedded value.

If the owner front phantom domain ID embedded in the entry is not avalid domain (less than 1 or greater than 239), it is ignored. If thelocal lowest front phantom domain ID is lower than the one embedded inthe entry, it is ignore. If the local lowest front phantom domain ID ishigher than the one embedded in the entry, set the local lowest domainID to the embedded value. If the local lowest front phantom domain ID isthe same as the embedded value in the entry, iterate through a list ofcached lowest domain ID, from all remote front phantom domains reachableto the edge remote fabric specified in the entry. If all cached lowestdomain IDs of all remote front phantoms are the same for the remote edgefabric, set the local owner front phantom domain ID to the matchingfront phantom domain ID.

If the local owner front phantom for the fabric specified in the entryis set:

If the incoming entry contains the same owner front phantom, this isnormal. If the incoming entry contains a different owner front phantom,this is an error case. The EX_port with higher domain ID is disabled andan error message is printed to notify users. After the first threeFCR_EF_ALIVE messages are transmit for a newly reachable edge fabric,iterate through the cached lowest and owner front phantom domain ID listto see if any other front phantom domains can reach the same fabric. Ifthe local front phantom domain is the only one able to reach to theremote fabric, set the local front phantom domain as the owner of thefabric.

The list of un-reachable edge fabric entries are handled based on thefollowing algorithm:

If the cached information of the remote front phantom domain, whichoriginated the HLO, was previously set to reachable to the nowun-reachable edge fabric, a full FSPF is sent to remove the link. If thelocal front phantom domain is able to reach the edge fabric and theremote front phantom domain is not able to but owns the edge fabric,clear out the voting information while saving the translate phantomdomain information from the remote front phantom domain. Also, send afull FSPF to reflect front phantom domain's ownership loss.

Devices connected to an edge fabric communicate with devices fromanother edge fabrics through proxy devices presented by EX_ports.However, when the Fibre Channel frames are delivered into a backbonefabric, source and destination addresses are converted to NR_ports fordelivery. These NR_ports can be a list of EX_ports connected to otheredge fabrics or special ports dedicated for fabric-to-fabric routing.Since the frame must be re-translated when arriving at an egress port,NR_port information is added as part of a special header to facilitatebackbone fabric 150 routing.

Each Fibre Channel router 100 maintains a full record of all FibreChannel routers 100 and all NR_ports in the global fabric. The globalfabric includes all edge fabrics and backbone fabrics. A routingdecision is calculated independently by each Fibre Channel router 100based on the full database of NR_ports. For certain embodiments eachFibre Channel router 100 maintains a full record of all Fibre Channelrouter 100 in the same backbone fabric 150 and the routing decision issimplified to use of a simple round-robin algorithm in case multipleNR_ports exist to a destination edge fabric. The algorithm includes theS_ID, D_ID, and OX_ID of the exchange. In other cases however, when thealgorithm only accounts for S_ID and D_ID. This operational model isvery similar to how FSPF works. Major differences are:

A Fibre Channel router 100 maintains only Fibre Channel routers 100 asnodes, not all switches in the path. However, within a backbone fabric150, path selection between NR_ports is done by FSPF.

A Fibre Channel router 100 maintains information outside of its localbackbone fabric. All Fibre Channel routers 100 in the global fabricshare and maintain the information in certain implementations.

A Fibre Channel router 100 maintains information based on a versionstamp, unlike FSPF which maintains a database based on aging andincarnation number.

The NR_port database is maintained based a on version stamp. Once anEX_port successfully establishes a connection to an edge fabric, theEX_port compares the fabric tag and Principal switch WWN of the edgefabric against a NR_port database. The NR_port database contains a listof Fibre Channel routers 100, which are called the router node entries.A router node contains a router node ID, which consists of the fabrictag and domain ID of the Fibre Channel router 100 the node represents; aversion stamp; and a list of NR_port entries. An NR_port entry containsthe NR_port's Port ID, destination fabric tag, Principal Switch WWN ofthe destination fabric and link cost. If the newly online EX_port'sfabric tag and Principal Switch WWN do not match existing information inthe database, the EX_port is disabled and an error message is providedto notify that the EX_port has been disabled because of the fabric tagconflict or over-subscription. If the newly online EX_port's informationmatches with the existing information in the database or suchinformation does not exist, a new NR_port is added with a currentversion stamp and updates are sent to all Fibre Channel routers 100 inthe global fabric.

When an EX_port receives a Fibre Channel frame that is destined foranother fabric, a destination domain ID is used to first identify thedestination fabric. Then a list of NR_port entries is searched todetermine available NR_ports for the destination edge fabric.

Certain implementations limit the number of router node entries perFibre Channel router 100 at 256 and the number of NR_port entries pernode at 256 and the cost is fixed at 1000, but other implementations canhave different limits.

Once an EX_port comes online, it sends out a FCR_BB_ALIVE at a shortfixed interval, such as five seconds, to all valid domains within abackbone fabric 150. The rate of FCR_BB_ALIVE messages sent is slowed,for example, to one minute over time unless other Fibre Channel routers100 come online within the backbone fabric 150. The FCR_BB_ALIVE messageis used to discover all Fibre Channel routers 100 within a backbonefabric 150 and to check to see if all of them have the same fabric tag.If a Fibre Channel router 100 contains a fabric tag that mismatches therest of the backbone fabric 150, it does not impede normal Fibre Channelrouter 100 operation for the switch. However, the Fibre Channel router100 is not able to route Fibre Channel frames through the backbonefabric 150 using other fabric-tag-mismatching Fibre Channel routers 100.

Once a new Fibre Channel router 100 with a matching fabric tag isdiscovered, the local Fibre Channel router 100 sends its own NR_portnode entry as a database update. The newly online Fibre Channel router100 responds with an acknowledgment. Anytime a router node entry thatbelongs to a Fibre Channel router 100 is updated, a database update issent to all known Fibre Channel routers 100 in the backbone fabric 150with a matching fabric tag. If no acknowledgment has been received and aFCR_BB_ALIVE message has been received from the switch, the databaseupdate is considered lost and the update is retransmitted. If noFCR_BB_ALIVE message has been received from a known Fibre Channel router100 while a Fibre Channel router 100 has sent out four FCR_BB_ALIVEmessages or a domain unreachable RSCN for the known Fibre Channel router100 has been received, the known Fibre Channel router 100 is consideredoffline and the router node entry is removed from NR_port database.

Once an EX_port successfully establishes a connection to an edge fabric,the EX_port is now ready for validation. A search is initiated through aNR_port database to see if the Principal switch WWN and fabric tag ofthe newly online EX_port conflicts with existing information in thedatabase. If there is a conflict, it means that the newly online EX_portis using the same fabric tag used by another edge fabric or the samefabric is taking up another fabric tag. In such case, the EX_port isdisabled to stop further participation and an error message provided.The port state of the EX_port is Reset. If there are no conflicts, theinformation is added to the NR_port database. Following is the algorithmto add the information to the database:

If the router node does not exist, a new router node is added based onthe local backbone fabric tag and the local Fibre Channel router 100domain ID. The entry contains zero NR_port entries and the router nodeentry is version stamped based on the local Fibre Channel router 100version stamp counter. A new NR_port entry is added if an entry does notalready exist for the new EX_port that came online. The number of theNR_port field is incremented by one. The fabric tag field is set to theedge fabric tag. The Port ID field is set to the local EX_port Port ID.The Cost field is fixed at 1000. The router node entry is versionstamped based on the local Fibre Channel router 100 version stampcounter. An update is sent to all Fibre Channel routers 100 in theglobal fabric.

Many of the functions performed by the Fibre Channel router 100according to the preferred embodiment a set of port processors 400 and acentral processor 408 (FIG. 12). Each port processor 400 can operate asboth an ingress port and an egress port. A crossbar switch 402 links theport processors 400. A control circuit 404 connects the central processor 408 to the crossbar switch 402 to allow the central processor 408 tocontrol the crossbar switch 402 and to provide a link to the portprocessors 400. Memory 406, which stores a set of executable programs,is connected to the central processor 408.

In particular, the memory 406 stores Fibre Channel connectivity software410; a management software 412; a set of industry standard applications414; lookup table development software 416, used to develop the lookuptables used in the port processors 400; front and translate phantomdomain software 418; EX_port and NR_port setup software 420 and softwareused to develop and handle any high level commands for the variousphantom or proxy devices.

It is also noted that a number of the functions associated with some ofthis software are partially performed on the port processors 400 inconjunction with the central processor 408. For example, the portprocessors 400 include hardware and software to work with the lookuptables to translate the addresses and develop the particular fabric IDsto allow the translation of frame addresses and development ofinterfabric and encapsulation headers.

FIG. 13 is a simplified illustration of a port processor 400. Each portprocessor 400 includes Fibre Channel and Gigabit Ethernet receive node430 to receive either Fibre Channel or IP traffic. The use of FibreChannel or Ethernet is selectable for each port processor. The receivenode 430 is connected to a frame classifier 432. The frame classifier432 provides the entire frame to frame buffers 434, preferably DRAM,along with a message header specifying internal information such asdestination port processor and a particular queue in that destinationport processor. This information is developed by a series of lookupsperformed by the frame classifier 432.

Different operations are performed for IP frames and Fibre Channelframes. Only Fibre Channel operations are described here For anexplanation of IP frame operation, please refer to U.S. patentapplication Ser. No. 10/695,625, entitled “Apparatus and Method forStorage Processor with Split Data and Control Paths”, filed Oct. 28,2003, which is incorporated by reference. For Fibre Channel frames theSID and DID values in the frame header are used to determine thedestination port, any zoning information, a code and a lookup address.The F_CTL, R_CTL, OXID and RXID values, FCP_CMD value and certain othervalues in the frame are used to determine a protocol code. This protocolcode and the DID-based lookup address are used to determine initialvalues for the local and destination queues and whether the frame is toby processed by the control module, an ingress port, an egress port ornone. The SID and DID-based codes are used to determine if the initialvalues are to be overridden, if the frame is to be dropped for an accessviolation, if further checking is needed or if the frame is allowed toproceed. If the frame is allowed, then the control module, ingress,egress or no port processing result is used to place the frame locationinformation or value in the embedded processor queue 436 for ingresscases, an output queue 438 for egress and control module cases or a zerotouch queue 439 for no processing cases. No interfabric frames would beplaced in the zero touch queue 439 because of the required addresstranslations and header operations. Generally control frames would besent to the output queue 438 with a destination port specifying thecontrol circuit 404 or would be initially processed at the ingress port.Fast path operations, such as frame switching, address translationand/or header development or removal, could use any of the three queues,depending on the particular frame.

Frame information in the embedded processor queue 436 is retrieved byfeeder logic 440 which performs certain operations such as DMA transferof relevant message and frame information from the frame buffers 434 toembedded processors 442. This improves the operation of the embeddedprocessors 442. The embedded processors 442 include firmware, which hasfunctions to correspond to some of the executable programs illustratedin memory 406 of FIG. 12. In the preferred embodiment three embeddedprocessors are provided but a different number of embedded processorscould be utilized depending on processor capabilities, firmwarecomplexity, overall throughput needed and the number of available gates.In various embodiments this includes firmware for determining andre-initiating SCSI I/Os; implementing data movement from one target toanother; managing multiple, simultaneous I/O streams; maintaining dataintegrity and consistency by acting as a gate keeper when multiple I/Ostreams compete to access the same storage blocks; performing addresstranslations; performing header development or removal; and handlingupdates to configurations while maintaining data consistency of thein-progress operations.

When the embedded processor 442 has completed ingress operations, theframe location value is placed in the output queue 438. A cell builder444 gathers frame location values from the zero touch queue 439 andoutput queue 438. The cell builder 444 then retrieves the message andframe from the frame buffers 434. The cell builder 444 then sends themessage and frame to the crossbar 402 for routing based on thedestination port value provided in the message.

When a message and frame are received from the crossbar 402, they areprovided to a cell receive module 446. The cell receive module 446provides the message and frame to frame buffers 448 and the framelocation values to either a receive queue 450 or an output queue 452.Egress port processing cases go to the receive queue 450 for retrievalby the feeder logic 440 and embedded processor 442. No egress portprocessing cases go directly to the output queue 452. After the embeddedprocessor 442 has finished processing the frame, the frame locationvalue is provided to the output queue 452. A frame builder 454 retrievesframe location values from the output queue 452 and changes any frameheader information based on table entry values provided by an embeddedprocessor 442. The message header is removed and the frame is sent toFibre Channel and Gigabit Ethernet transmit node 456, with the framethen leaving the port processor 400.

The embedded processors 442 thus include both ingress and egressoperations. In the preferred embodiment, multiple embedded processors442 perform ingress operations, preferably different operations, and atleast one embedded processor 442 performs egress operations. Theselection of the particular operations performed by a particularembedded processor 442 can be selected using device options and theframe classifier 432 will properly place frames in the embeddedprocessor queue 436 and receive queue 450 to direct frames related toeach operation to the appropriate embedded processor 442. In othervariations multiple embedded processors 442 will process similaroperations, depending on the particular configuration

As previously indicated, various functions are distributed between theport processor 400 and the central processor 408. Each port processor400 implements many of the required functions. This distributedarchitecture is more fully appreciated with reference to FIG. 14. FIG.14 illustrates the implementation of the Fibre Channel connectivitysoftware 410. As shown in FIG. 14, the control processor 408 implementsvarious functions of the Fibre Channel connectivity software 410 alongwith the port processor 400.

In one embodiment according to the invention, the Fibre Channelconnectivity software 410 conforms to the following standards: FC-SW-2fabric interconnect standards, FC-GS-3 Fibre Channel generic services,and FC-PH (now FC-FS and FC-PI) Fibre Channel FC-0 and FC-1 layers.Fibre Channel standard connectivity is provided to devices using thefollowing: (1) F_port for direct attachment of N_port capable hosts andtargets, (2) FL_port for public loop device attachments, and (3) E_portfor switch-to-switch interconnections. Selected EX_port and NR_portconnectivity is handled in this software.

In order to implement these connectivity options, the apparatusimplements a distributed processing architecture using several softwaretasks and execution threads. FIG. 14 illustrates tasks and threadsdeployed on the control module and port processors. The data flow showsa general flow of messages.

An FcFrameIngress task 500 is a thread that is deployed on a portprocessor 400 and is in the datapath, i.e., it is in the path of bothcontrol and data frames. Because it is in the datapath, this task isengineered for very high performance. It is a combination of portprocessor core, feeder queue (with automatic lookups), andhardware-specific buffer queues. It corresponds in function to a portdriver in a traditional operating system. Its functions include: (1)serialize the incoming fibre channel frames on the port, (2) perform anyhardware-assisted auto-lookups, particularly including frameclassification, (3) queue the incoming frame, and (4) develop or removeany additional headers.

Most frames received by the FcFrameIngress task 500 are placed in theembedded processor queue 436 for the FcFlowIngress task 506. However, ifa frame qualifies for “zero-touch” touch” option, that frame is placedon the zero touch queue 439 for the crossbar interface 504. The framemay also be directed to the central processor 408 in certain cases.These cases are discussed below. The FcFlowIngress task 506 is deployedon each port processor in the datapath. The primary responsibilities ofthis task include:

1. Dispatch any incoming Fibre Channel frame from other tasks (such as,FcpNonRw) to an FcXbar thread 508 for sending across the crossbarinterface 504.

2. Allocate and de-allocate any exchange related contexts.

3. Perform any Fibre Channel frame translations.

4. Update usage and related counters.

5. Develop any interfabric or encapsulation headers.

The FcXbar thread 508 is responsible for sending frames on the crossbarinterface 504. In order to minimize data copies, this thread preferablyuses scatter-gather and frame header translation services of hardware.This FcXbar thread 508 is performed by the cell builder 444.

Frames received from the crossbar interface 504 that need processing areprovided to an FcFlowEgress task 507. The primary responsibilities ofthis task include:

1. Perform any Fibre Channel frame translations.

2. Update usage and related counters.

If no processing is required or after completion by the FcFlowEgresstask 507, frames are provided to the FCFrameEgress task 509. Essentiallythis task handles transmitting the frames and is primarily done inhardware, including the frame builder 454 and the transmit node 456.

An FcpNonRw task 510 is deployed on the central processor 408. Theprimary responsibilities of this task include:

1. Analyze FC frames that are not Read or Write (basic link service andextended link service commands).

2. Keep track of error processing.

A Fabric Controller task 512 is deployed on the central processor 408.It implements the FC-SW-2 and FC-AL-2 based Fibre Channel services forframes addressed to the fabric controller of the switch (D_ID 0xFFFFFDas well as Class F frames with PortID set to the DomainId of theswitch). The task performs the following operations for each domain inthe Fibre Channel router 100, either real or phantom:

1. Selects the principal switch and principal inter-switch link (ISL).

2. Assigns the domain id for the switches.

3. Assigns an address for each port.

4. Forwards any SW_ILS frames (Switch FSPF frames) to the FSPF task.

A Fabric Shortest Path First (FSPF) task 514 is deployed on the controlmodule 202. This task receives Switch ILS messages from theFabricController 512 task. The FSPF task 514 implements the FSPFprotocol and route selection algorithm. It also distributes the resultsof the resultant route tables to all exit ports of the switch.

An FcNameServer task 518 is also deployed on the control module 202.This task implements the basic Directory Server module as per FC-GS-3specifications. The task receives Fibre Channel frames addressed to0xFFFFFC and services these requests using the internal name serverdatabase. This database is populated with Initiators and Targets as theyperform a Fabric Login. Additionally, the Name Server task 518implements the Distributed Name Server capability as specified in theFC-SW-2 standard. The Name Server task 518 uses the Fibre Channel CommonTransport (FC-CT) frames as the protocol for providing directoryservices to requestors. The Name Server task 518 also implements theFC-GS-3 specified mechanism to query and filter for results such thatclient applications can control the amount of data that is returned.

A management server task 520 implements the object model describingcomponents of the switch. It handles FC Frames addressed to the FibreChannel address 0xFFFFFA. The task 520 also provides in-band managementcapability. The module generates Fibre Channel frames using the FC-CTCommon Transport protocol.

A zone server 522 implements the FC Zoning model as specified inFC-GS-3. Additionally, the zone server 522 provides merging of fabriczones as described in FC-SW-2. The zone server 522 implements the “SoftZoning” mechanism defined in the specification. It uses FC-CT CommonTransport protocol service to provide in-band management of zones.Further discussion on multiple fabric zoning is provided below withrespect to FIG. 20.

A CMConfig task 524 performs the following operations:

1. Maintain a consistent view of the switch configuration in itsinternal database.

2. Update ports in I/O modules to reflect consistent configuration.

3. Update any state held in the I/O module.

4. Update the standby control module to reflect the same state as theone present in the active control module.

As shown in FIG. 14, the CMConfig task 524 updates a PortConfig task526. The PortConfig task 526 is a thread deployed on the port processor400. The task 524 performs the following operations:

1. Update of any configuration tables used by other tasks in the portprocessor, such as the frame classification and access control tables.This update shall be atomic with respect to other ports.

2. Ensure that any in-progress I/Os reach a quiescent state.

The FcpNonRW task 510 forwards certain frames to various other tasks forspecialized functions related to the routing capabilities of the FibreChannel router 100. A first task is the EX_port and NR_port setup task528. While the lower level capabilities and initialization of EX_portsand NR_ports are performed by initialization software based onoperator-defined values indicating a particular port is anEX_port/NR_port, higher level functions are performed by the setup task528 when devices that may use the specific port are identified. Thus thesetup task cooperates with a lookup table development task 532 and afront and translate phantom task 534. For example, the NR_port discoverydescribed above is handled in the setup task 528.

The lookup table development task 532 provides the necessary entries inthe port processor mapping tables. Entries are provided as devices andfabrics are connected or removed. The entries provide the informationnecessary to perform address translations and interfabric andencapsulation header development.

The front and translate phantom task 534 receives the frames directed tohigher level operations, such as management and standards-based, for thefront and translate phantom domains. Many of these operations aredescribed above. These operations include:

1. Fabric Services for the phantom domains.

2. ILS handling.

3. Port management.

4. Other EX_port and NR_port tasks.

An additional task is the proxy device handling task 530. Proxy devicesare treated as being connected to the translate phantom domain by loops.As described above, the translate phantom domains can use ports 0 to 238to connect to front phantom domains (there being a maximum of 239domains in fabric), ports 239 to 255 are available to be used for proxydevices. In the preferred embodiment each loop can contain 127 AL_PAs ordevices for simplicity, through 255 AL_PAs could be used if defined.Therefore the preferred embodiment allows the connection of 2,159devices for interfabric communication, though 4,335 devices could beused if all 255 AL_PAs are used. The proxy handling task 530 cooperateswith the zone server task 522 to assign addresses to the proxy devicesand with the lookup table development task 532 to incorporate thesedevice addresses into the table.

It is understood that the tasks 512, 518, 520 and 522 can operatecentrally for all relevant domains, such as those of the Fibre Channelrouter 100 acting as a level two switch and the various phantom domains,or multiple instances can be used to handle different groups of domains,such as the Fibre Channel router 100 itself and the various phantomdomains, or can handle each domain individually. Thus the tasks shown inthe central processor 408 are to illustrate the functions to beperformed and the actual instances of relevant software can be varied asdesired.

Referring then to FIG. 15, this is a flow chart of the operation of theARM processor or embedded processor 442 of an EX_port. For a furtherexplanation of the operation of embedded processors 442 and division oftasks between fast path and control or slow path, refer to applicationSer. No. 10/695,625 referenced above. In Step 700 a frame is receivedwhich is directed to a proxy device. In step 702 the hardware in theport processor 400 retrieves the final DID and the destination fabricvalues and in step 704 the embedded processor performs a hash on theSID, DID and OXID, which allows exchange-based load balancing, andperforms a table lookup to obtain the destination port in the FibreChannel router 100. In step 706 the embedded processor 442 develops thevarious port IDs and translates the frame header entries needed tocorrectly address the source and destination devices according to thetranslate phantoms present and the particular end of frame and port IDsnecessary to traverse the backbone 150. In step 708 the processor buildsthe fabric routing header and the encapsulation header and appends thoseto the packet. In step 710 the frame is sent from the port processor 400for routing to the receiving port or egress port.

FIG. 16 is a similar flowchart for embedded processor 442 operation atthe NR_port. There are preferably multiple embedded processors 442 ineach port processor 400 so that in step 800 the embedded processor 442receives the frame from the remote fabric i.e. the frame that had beendeveloped by the EX_port processor. In step 802 the embedded processor442 performs a proxy source ID look-up based on the source fabric ID andreal SID and provides this information to the second embedded processor442 in the port processor 400. In step 804 the embedded processor 442removes the fabric routing header and encapsulation header and then instep 806 translates the frame header entries as necessary based on theproxy device and target device. Then in step 808 it sends the frame intothe destination fabric for routing to the destination device. Thus theoperations utilized to perform the translations and headers developmentand removal are relatively simplistic and can be performed at full wirespeed.

It is noted that the EX_port uses ARM1 and the NR_port uses ARM 2. Thusthe port can readily be both an EX_port and an NR_port in the preferredembodiment because different embedded processors are used.

An alternate embodiment is shown in FIG. 17. Fibre Channel routers 900are interconnected to allow connection of two fabrics. In theillustrated embodiment a first Fibre Channel router A 900A connected viaEX_ports to a Fibre Channel router B 900B. Fibre Channel router A 900Ais connected to a fabric A 102A through E_ports to switches and F_portsto nodes. Similarly the Fibre Channel router B 900B is connected to afabric B 102B through F_ports and E_ports to the various devices in thefabric B 102B. Routing between fabrics 102A, 102C and 102D would behandled by Fibre Channel router A 900A, while between fabrics 102/B,102E and 102F would be handled by Fibre Channel B 900B. Routing from anyof fabrics 102A, 102C, 102B, 102E or 102F would be done using both FibreChannel routers A and B 900A and 900B. This configuration providesadditional fault isolation between the connected fabrics but alsorequires an additional Fibre Channel router.

FIG. 18 illustrates that the Fibre Channel routers A and B 900A and 900Bcan actually connect to multiple fabrics such as 102A, 102C and 102D inthe illustrated case for Fibre Channel router A 900A and fabrics 102B,102E and 102F for Fibre Channel router B 900B.

FIG. 19 is a fabric interconnection illustration using this particularinterconnection of the Fibre Channel routers for a more complicatedenvironment.

The primary method for managing inter-fabric device sharing is via WorldWide Name (WWN) zoning. In this scheme, users create zones at each edgefabric using normal zoning management interfaces available at each edgefabric. Thus no changes to the zoning interfaces present in standardFibre Channel devices are required. However, for inter-fabric devicesharing, the inter-fabric zones have a special name according to thefollowing convention: they are named “LSAN_XYZ”, where XYZ represents auser-defined string. The LSAN tag in the zone name indicates to theFibre Channel router that the zone is a special, inter-fabric zone.

The WWNs of the devices being used in inter-fabric device sharing arethe elements of these special zones, known as “LSAN Zones”. The WWNscontained in these zones may exist on the local edge fabric (the localedge fabric where the zone is created) or may exist in another fabric.WWNs in the LSAN zones that exist in the local edge fabric are devicesintended for export to another fabric, and WWNs that exist in anotherfabric represent devices intended for import into this local edgefabric.

Fibre Channel routers 100 obtain the zoning database from each edgefabric and parse the database for zones with the special “LSAN_” tag inthe zone name. Each LSAN zone database entry consists of the LSAN zonename, a list of port WWNs, and a device state record. The special prefixis preferably case insensitive so “LSAN_” and “lsan_” are both valid, aswell as other capitalization schemes. The Fibre Channel router 100compares the WWNs from the special zones from a particular fabric withentries from another fabric. Matching entries define which devices cancommunicate via Fibre Channel router(s) 100 across fabrics. Thesedevices are considered to be in the same LSAN.

The LSAN zone database and device database are preferably organized asfixed size arrays. However, to support fast lookup of device entries andfinding of two zoned entries in another fabric, hash tables of thedevice database organized based on port WWN and on fabric ID combinedwith port WWN are implemented. The hash implementation allows the FibreChannel router 100 to handle a large number of device state changes andspeeds up import/export checking.

The LSAN zone database maintained by the Fibre Channel router 100 breaksdown into two different entry types: edge LSAN zone entries and backboneLSAN zone entries. Edge LSAN zone entries are created based on edgefabrics' active zone entries, and may be further divided into twoadditional types: local and remote. Local edge LSAN zone entries arecreated based on edge fabrics directly connected to a Fibre Channelrouter 100. Remote edge LSAN zone entries are cached entries from otherFibre Channel routers' local entries. Backbone LSAN zone entries arecreated based on a backbone fabrics' active zone entries and may also befurther divided into local and remote types. Local backbone LSAN zoneentries are created based on backbone fabrics directly connected to theFibre Channel router, while remote backbone LSAN zone entries are cachedof entries from other backbones.

Local edge LSAN zone entries are created through zone operations betweenan edge fabric and a front phantom. When an edge fabric connects to anEX_Port, it sends the full zoning database as part of “merging” processwith a front phantom created by the EX_Port. The zone entries with namesthat start with the special tag are converted to local LSAN zoneentries. Local LSAN zone entries are maintained based on othersubsequent zone operations, such as create, delete, etc., initiated bythe edge fabric.

Remote edge LSAN zone entries are created through Fibre Channel routers100 connected to the same backbone exchanging local edge LSAN zoneentries. Once Fibre Channel routers 100 discover one another, theyinitiate full database exchange and subsequently follow with updates tomaintain a completely synchronized database across all Fibre Channelrouters 100 within the same backbone. Updates are sent with the fullLSAN zone entry and are acknowledged by another Fibre Channel router 100with LSAN zone name and version stamp. Backbone LSAN zone entries areexchanged between backbone fabrics through a Fibre Channel router 100protocol. The exchange involves initial LSAN zone database exchange whentwo EX_Ports are established, then subsequently followed by updates.

The LSAN zone databases are used by the Fibre Channel router 100 todetermine import/export requirements for mapping phantom devices intothe external fabric with which they are communicating. The import/exportcondition is quite simple. When two fabrics contain a matching LSAN zonepair, and each fabric contains one of two devices, the Fibre Channelrouter 100 imports/exports each device to the remote fabrics. Forexample, fabric A contains device DA and fabric B contains device DB.Fabric A contains an LSAN zone entry that allows DA and DB to talk toeach other, and fabric B contains an LSAN zone entry that allows thesame. Once it has been verified that both fabrics allow two devices tocommunicate with each other and both devices are online in theirrespective fabrics, the Fibre Channel router 100 imports/exports deviceDB to fabric A and device DA to fabric B.

Better understanding of this scheme may be understood with reference toFIG. 20, which contains four fabrics: fabric A 102A, fabric B 102B,fabric C 102C, and fabric D 102D. The fabrics are interconnected byFibre Channel router FCR100A. Fabric A 102A contains host 104.Similarly, fabric B 102B and fabric D 102D contain host 106 and storagedevice 110, respectively. Fabric C 102C contains a storage device 108.In fabric A 102A, an LSAN zone LSAN_fabric A is defined that includeshost 104, host 106, and storage device 110. Similarly, in fabric C 102C,an LSAN zone LSAN_fabricC is defined that also contains host 104, host106, storage device 108 and storage device 110. The Fibre Channel router100A connecting these two zones will compare the contents of the LSANzones and will allow communication between host 104 and storage device108 because these elements are common to the defined LSAN zones.Conversely, in fabric B 102B, an LSAN zone is defined that includes onlyhost 106, storage device 108 and storage device 110. Thus host 106 willbe able to communicate with storage device 106 in fabric C 102C, buthost 106 will not be allowed to communicate with host 104 in fabric A102A because there is no overlapping LSAN zone that includes hosts 104and 106.

It is important to note several things about this zoning scheme. First,the user defined (i.e., “XYZ”) portion of the LSAN zone name from onefabric does not have to match the user-defined portion of the LSAN zonename from another fabric for devices to be included in the same LSAN.Hence LSAN_fabricA and LSAN_fabricC allow communication between theidentified components. Second, this zoning scheme is intended to workwith real zones in the edge fabrics and the devices that exist in thesezones are subject to normal zoning enforcement by the switches in thefabric. Additionally, LSAN zones must be World Wide Name zones andcannot be any other type of zone (e.g., port zones, etc.) Finally, nospecial firmware (assuming WWN zoning is supported) is required in theedge fabrics to enable this inter-fabric device sharing managementmodel.

The zoning example discussed above is advantageous in a network whereseparate fabrics are each administered by a different administrator.This allows the two administrators to configure inter-fabriccommunication without releasing control of any aspects of their ownfabric. Conversely, in the case where a single administrator has controlof all of the edge fabrics, the Super Administrator model provides thecapability for a single administrator (i.e., the Super Administrator) tocentrally administer the LSANs using a single step process. This ispreferably accomplished by a management application that implements aGUI allowing the Super Administrator to drag and drop devices into anLSAN. This management application will then automatically setup LSANzones in the various fabrics in a behind the scenes process that istransparent to the user. One such management application which couldimplement this process is Fabric Manager from Brocade Communications.

LSAN zone entries must remain synchronized among Fibre Channel routers100 to facilitate routing across backbone fabrics 150 and to gracefullyrecover from Fibre Channel router 100 failures. The protocol forsynchronization within a backbone fabric 150 allows for all FibreChannel routers 100 within a backbone to maintain the latest LSAN zoneentries that a Fibre Channel router 100 does not have direct connectionto. This allows two Fibre Channel routers 100 within a backbone tosynchronize import/export across the backbone fabric 150. Every 5seconds, a Fibre Channel router daemon runs through local edge LSAN zoneentries to see if any entries must be shared with other Fibre Channelrouters 100 within a backbone fabric 150. Local edge LSAN zone entriesare sent only if: (1) the remote Fibre Channel router 100 belongs to thesame backbone fabric as the local Fibre Channel router 100, (2) theremote Fibre Channel router 100 is not directly connected to an edgefabric that created the entries, and (3) the last acknowledge messagefrom the remote Fibre Channel router 100 regarding the LSAN zone entryis older than the current LSAN zone timestamp.

The protocol for synchronization within an edge fabric protocol allowsthe owner front phantom to maintain a consistent view of deviceinformation if the device belongs to an edge fabric that the frontphantom owns. As an owner front phantom maintains a translate domain andnode WWN consistent within an edge fabric, the owner front phantom isalso responsible for maintaining device information, mainly proxy PortID and device state, within an edge fabric. Every 10 seconds, each FibreChannel router 100 through local edge LSAN zone entries created by theedge fabric the front phantom is connected to. When it finds a devicethat imported/exported into a fabric the front phantom owns, the frontphantom checks to see if the device information needs to be shared withother front phantoms connected to the same edge fabric. The deviceinformation is sent only if the other front phantom is also reachable tothe remote fabric where the device resides in and the last ACK from theother front phantom regarding the device is older than the timestamp ofthe device maintained by the front phantom sending the deviceinformation.

As noted above, routing between fabrics requires additional headers tothose specified by the Fibre Channel standard. To accomplishinter-fabric routing, a “global header” is inserted into frames toprovide Fibre Channel routers 100 in a frame's path with instructions onthe source and destination fabric, as well as the source device port IDin the source fabric and the destination device port ID in thedestination fabric. This involves adding this global header to FibreChannel frames to allow unique addressing of devices across multipleinterconnected fabrics. A local header, i.e., a normal Fibre Channelheader, is used to route within a fabric.

The global header 2100, illustrated in FIG. 21, contains a timestamp2101 and global port identifier fields (GPIDs) that uniquely identifythe source (GSID 2102) and destination (GDID 2103) ports of the frame ina connected network of distinct (i.e., not merged) fabrics. In certainembodiments, end devices will insert GPIDs into the global header of aframe to uniquely address devices and switches will use the GPIDs inconjunction with normal (local fabric) PIDs to make routing decisions.Alternatively, for legacy devices that are not capable of globaladdressing, the Fibre Channel router 100 will have the capability tocreate the global header on behalf of the legacy devices that are notcapable of global addressing.

The timestamp included in the global header indicates the expirationtime of a frame. The format of the timestamp could take on a variety offorms, each of which has various advantages and disadvantages, whichwould be understood by one of ordinary skill in the art. Switches in thedata path compare the timestamp with the current time and discard theframe if the current time is beyond the timestamp. Obviously, thisrequires all switches that enforce the timeout to have synchronizedtime. Time synchronization is not viewed as problematic because mostswitches support NTP time synchronization and the synchronizationaccuracy demands are not very high given that the timeouts are expectedto have granularities of seconds or minutes. It is expected that themaximum timeout value will be less than 2 minutes, although other valuesof the same order of magnitude would also be feasible. Some switches orFibre Channel routers 100 in the frame's path may check this timestampand others may not (legacy switches). This frame level timer granularityallows much more flexibility to build complex multi-fabric SANs.

Each GPID (global port identifier field) has two subfields. The upper 16bits of the GPID will identify a fabric, source fabric (SFABRIC 2104)for a GSID and destination fabric (DFABRIC 2105) for a GDID. The lower24 bits of the GPID contain the end device local PID, i.e., Local SID2106 and Local DID 2107. The local PID sub-field in a GSID contains theSID of the end device sending a frame. This SID is the originating enddevice's PID. This PID is relevant to the local fabric of theoriginating end device. The local PID sub-field in a GDID contains theDID of the end device that is to receive the frame. This DID is thereceiving end device's PID. This PID is relevant to the local fabric ofthe receiving end device.

Various locations within a Fibre Channel frame are possible for theglobal header. In one embodiment, the global header 2201 is inserted inthe Fibre Channel frame between the start of frame indicator and theframe header 2204. Such a frame is illustrated in FIG. 22. In anotherembodiment, the global header can be a new Fibre Channel optional headerimmediately following the standard Fibre Channel header and prior to theframe's payload. In this embodiment, the global header 2201 resides inthe data portion 2202 of the Fibre Channel frame 2200 along with theframe's payload 2203. Its presence is indicated by DF_CTL[23] in theFibre Channel frame header 2204, which is currently a reserved bit. Ifother optional headers are present (e.g., ESP header 2205, networkheader 2206, etc.), they will follow the global header if present.

As an alternative to including the global header in the data portion ofa Fibre Channel frame, the global header can be an extension to theFibre Channel frame header (2204, FIG. 22) using R_CTL bits (first fieldof an FC frame header, described below with reference to FIG. 24) tospecify a new frame format. The R_CTL bit format that would specify thepresence of the global header could be any of a number of possibilities,all of which could be conceived by one of ordinary skill in the art.This frame format would extend the Fibre Channel frame header size bythe length of the global header. The global header would then reside atthe end of the frame header. The portion of the header preceding theglobal header would remain the same format as it is currently topreserve backward compatibility as much as possible. This extended frameheader option does not occupy data field bytes, thus allowing more roomfor payload and overcoming the problems of data field use discussedabove.

In the embodiment where the optional header shares the data portion ofthe frame with the payload, the presence of an optional headereffectively reduces the size of the payload. In the current FibreChannel standard, the data field is limited to 2112 bytes. Thislimitation may cause an end device to require more frames to send astream of data. Although it is anticipated that future Fibre Channeldevices will be capable of performing global addressing on their own, tosupport legacy devices, the Fibre Channel router 100 will create theglobal header on behalf of the legacy devices that are not capable ofglobal addressing. In such a case, there may not be sufficient space inthe data field for the optional header if the frame's payload approaches2112 bytes in length.

One solution to this problem is for the Fibre Channel router 100 tointercept any special frames (e.g., PLOGI) that negotiate payload sizebetween the end device and the proxy device and modify the payload sizeto ensure sufficient space in the data field for the global header. Analternate solution is for the Fibre Channel router 100 to break theframe into two smaller frames.

The normal Fibre Channel header of the frame contains SID and DIDfields. In the originating fabric, the SID is the PID of the end devicesending a frame. For an originating end device that is global-headercapable, the DID specifies phantom port that appears to reside onoriginating device's local fabric. This DID effectively provides routinginstructions to a fabric to get frames from the originating device tothe Fibre Channel router's translation engine. If the end device is alegacy non-global-header capable device, then the DID represents aphantom device created by a Fibre Channel router 100, which is a proxyfor a device on a remote fabric. This DID also effectively providesrouting instructions to a fabric to get frames from the originating enddevice to a Fibre Channel router's translation engine.

If the destination end device is global-header capable, the SID is thePID of the Fibre Channel router translation engine, and the DID is thePID of the destination end device that is to receive a frame. If thedestination end device is a legacy non-global-header capable device thenthe SID represents a proxy device created by the Fibre Channel router100, which is a proxy for the originating device on a remote fabric.

For interfabric routing between fabrics connected by a backbone fabric(e.g., network topologies as in FIG. 5), different, and more, headersare required. The Fibre Channel router 100 described herein requires twonew frame headers, the encapsulation header and the fabric routingheader and further includes an additional normal header in the frame.These headers are used to pass frames between Fibre Channel routers 100that are connected to the same backbone fabric 150. Specifically, framesare passed between “NR_Ports” that are exposed by the Fibre Channelrouters 100, as described above. These headers are inserted by theingress EX_Port handler and are interpreted and removed by the egressNR_Port. Although various formats for these headers are possible, oneexemplary embodiment is described below. Nonetheless, in a preferredembodiment of the Fibre Channel router 100, frame handling is performedin software, so that format changes can be easily accommodated withoutrequiring major system changes.

The fabric routing header provides Fibre Channel routers 100 in the datapath with useful information that is used for routing and proxy devicecorrelation. FIG. 23 illustrates one proposed layout of the fabricrouting header 2300. Fabric routing header is 64 bits in length, and iscomprised of two 32 bit words 2301 and 2302. In the first field 2303 ofthe first word 2301 of the header (the R_CTL field), a set value of 0x51h indicates that the header is a fabric routing header. The final 16 bitfield 2304 of first word 2301 is the destination fabric identification(DF_ID) field. The remaining portions of the first word comprise a Verfield, set to zero, and six reserved bits divided into three fields2306. The second word 2302 comprises an expiration time field 2307,which can be set to zero to specify no expiration time. The final 16bits of the second word is the source fabric identification (SF_ID)field 2308. The SFID field is used by the egress port, in conjunctionwith the SID field in the normal Fibre Channel frame header to derivethe source proxy ID to use when sending the frame into the destinationedge fabric. The remaining 8 bits of the second word are reserved andare divided into two fields 2306.

The encapsulation header is used to encapsulate the Fibre Channel framesent from the source device to the destination device as well asencapsulating the global header. The encapsulation header is formattedexactly like a normal Fibre Channel header to facilitate transportinginter fabric frames through legacy fabrics with legacy switches. Becauseframes are sent from NR_Port to NR_Port across a backbone fabric, theaddressing in the header (SID/DID) represents the source NR_Port anddestination NR_Port. Details on the encapsulation header are illustratedin FIG. 24.

Encapsulation header 2400 comprises six words 2401-06. First word 2401contains two fields, first field R_CTL is set to 0x04, indicating thatthe frame contains unsolicited data. The second field of word 2401 isthe destination identifier (DID), which is set to the port identifier ofthe destination NR_Port. The second word 2402 is also divided into twofields. The first, CS_CTL, is copied from the original frame header. Thesecond field of the second word is the source identifier, i.e., the PortID of the source EX_Port. The third word 2403 is also divided into twofields, the first of which is a type field. Preferably this field is setto a particular value, e.g. 0x59, to indicate that the frame is part ofan “encapsulation tunnel.” The second field of word 2403 is F_CTL, whichis copied from the original header.

The fourth word 2404 of encapsulation header 2400 includes three fields.The three fields, SEQ_ID, DF_CTL, and SEQ_CNT, are copied from theoriginal Fibre Channel header. The fifth word 2405 also includes twofields, OX_ID and RX_ID, which are also copied from the original FibreChannel header. Finally, the sixth word 2406 includes a single field,Parameter, which is also copied from the original Fibre Channel header.

The normal Fibre Channel header format is also illustrated in FIG. 24(as noted above, the format of the encapsulation header is identical tothe format of a normal Fibre Channel header. A version of a normalheader is provided in the frame as well. This header contains the trueor local SID and the DID values of the devices and the remaining fieldsare copied from the original normal header received by the EX_port.

The following describes the contents of these fields at key locations inthe data path as a frame moves from a source in a first fabric to adestination in a second fabric wherein the two fabrics are connected viaa backbone fabric. An example of such a path can be envisioned withreference to FIG. 9, in which a frame originating at host 104 destinedfor storage device 108 passes through fabric A 102A to Fibre Channelrouter A 100A, into the backbone fabric 150. From the backbone fabric150, the frame enters second Fibre Channel router B 100B and passes intofabric B 102B ultimately arriving at destination storage device 108.

As the frame leaves host 104, it passes through fabric A 102A to anEX_Port of Fibre Channel router A 100A. At the ingress to Fibre Channelrouter A 100A, the relevant fields of the Fibre Channel Header are asfollows: DID is the Port ID of phantom device 108: the proxy device thatrepresents storage device 108 in fabric A 102A. This is a valid port IDin Fabric A. SID is the Port ID of the physical source device, i.e.,host 104. This is the valid, true or local ID host 104 in Fabric A. TheFibre Channel router A 100A does not utilize any other fields at thisstage. The frame is passed by Fibre Channel route A 100A in the backbonefabric 150

In backbone fabric 150, the normal encapsulation header indicates thesource and destination NR_ports and is used by any switches in thebackbone 150 for normal routing. The next header in the frame is thefabric routing header, which contains the source and destination fabricIDs. The final header in the frame is a copy of the original normalheader with one moidification. While originally the DID was the proxydevice, in this use the true or local DID value is used so that theheader contains the true DID and SID values for each end device.

After traversing the backbone fabric 150, the frame arrives at FibreChannel router B 100B, which removes the encapsulation and fabricrouting headers and provides a new header as it passes the frame intofabric B 102B. The final header is retrieved from the frame to form thenew header and the proxy SID value replaces the true SID value. The DIDremains unchanged from its previous state, and thus this field continuesto contain the Port ID of the physical destination device (storagedevice 108) in fabric B 102B). The SID is modified by Fibre Channelrouter B 100B to the Port ID of the proxy source device 104, i.e., therepresentation of host 104 as it appears in fabric B 102B. This is avalid Port ID in Fabric B 102B. No other fields are used at this stage.

Thus it has been shown that a Fibre Channel router has been developedwhich allows interconnection of separate Fibre Channel fabrics withoutrequiring that the fabrics merge. The Fibre Channel router uses anEX_Port to connect to an edge fabric, the EX_Port having properties thatit is included in the fabric but the router itself is not merged intothe fabric. The Fibre Channel router provides proxy devices and domainsto allow nodes on edge fabrics to address remote nodes as though theywere on the edge fabric. The Fibre Channel router handles any necessaryconversions. Part of this task includes adding various headers,including global, used to provide network wide addressing by identifyingsource and destination fabrics; interfabric headers, used to providesimplified addressing of the various Fibre Channel router portsconnected to the edge fabrics; and encapsulation headers, used to allowlegacy switches to route the frames through a backbone fabric which canconnect two Fibre Channel routers. Interfabric zoning is accomplished bydesignating a special zoning prefix of LSAN used in naming zones. EachFibre Channel router exchanges these LSAN zones and imports or exportsdevices according to intersections of the zones. These various elementsallow development of larger SANs and allow simplified operation of thoseSANs.

While illustrative embodiments of the invention have been illustratedand described, it will be appreciated that various changes can be madetherein without departing from the spirit and scope of the invention.

1. A carrier signal modulated with an augmented Fibre Channel frame thatcomprises: a start of frame field; a global header field for interfabricrouting; a frame header; a data field; and an end of frame field
 2. Thecarrier signal of claim 1 wherein the global header field comprises aglobal source identifier.
 3. The carrier signal of claim 1 wherein theglobal header field comprises a global destination identifier.
 4. Thecarrier signal of 2 wherein the global header field also comprises aglobal destination identifier.
 5. The carrier signal of claim 2 whereinthe global source identifier comprises a source fabric identifier thatidentifies an originating fabric of the frame and a source deviceidentifier that identifies an originating device of the frame within theoriginating fabric.
 6. The carrier signal of claim 3 wherein the globaldestination identifier comprises a destination fabric identifier thatidentifies a target fabric of the frame and a destination deviceidentifier that identifies a target device of the frame within thetarget fabric.
 7. The carrier signal of claim 4 wherein the globalsource identifier comprises a source fabric identifier that identifiesan originating fabric of the frame and a source device identifier thatidentifies an originating device of the frame within the originatingfabric and wherein the destination fabric identifier comprises adestination fabric identifier that identifies a target fabric of theframe and a destination device identifier that identifies a targetdevice of the frame within the target fabric.
 8. The carrier signal ofclaim 1 wherein the global header field comprises a timestamp indicatingan expiration time of the frame.
 9. The carrier signal of claim 4wherein the global header field comprises a timestamp indicating anexpiration time of the frame.
 10. The carrier signal of claim 7 whereinthe global header field comprises a timestamp indicating an expirationtime of the frame.
 11. The carrier signal of claim 1 wherein the globalheader field is part of the data field.
 12. A carrier signal modulatedwith an augmented Fibre Channel frame that comprises: a start of framefield; a frame header; a data field; and an end of frame field; whereinthe frame header comprises a global header field for interfabricrouting.
 13. The carrier signal of claim 12 wherein the global headerfield comprises a global source identifier and a global destinationidentifier.
 14. The carrier signal of claim 13 wherein the global sourceidentifier comprises a source fabric identifier that identifies anoriginating fabric of the frame and a source device identifier thatidentifies an originating device of the frame within the originatingfabric and wherein the destination fabric identifier comprises adestination fabric identifier that identifies a target fabric of theframe and a destination device identifier that identifies a targetdevice of the frame within the target fabric.
 15. The carrier signal ofclaim 12 wherein the global header field comprises a timestampindicating an expiration time of the frame.
 16. The carrier signal ofclaim 13 wherein the global header field comprises a timestampindicating an expiration time of the frame.
 17. The carrier signal ofclaim 14 wherein the global header field comprises a timestampindicating an expiration time of the frame.
 18. A method for routingFibre Channel packets between a first device on a first fabric and asecond device on a second fabric, the method comprising: receiving aFibre Channel packet from the first device, wherein a frame header ofthe packet includes a source identifier corresponding to the firstdevice in the first fabric and a destination identifier corresponding toa phantom representation in the first fabric of the second device;changing the destination identifier of the packet to correspond to thesecond device in the second fabric; and passing the packet to the secondfabric.
 19. The method of claim 18 further comprising: receiving thepacket in the second fabric; and changing the source identifier of thepacket to correspond to a phantom representation in the second fabric ofthe first device.
 20. The method of claim 18 further comprising: addinga global header to the Fibre Channel packet comprising a global sourceidentifier and a global destination identifier.
 21. The method of claim20 further comprising: changing the source identifier of the packet tocorrespond to a phantom representation in the second fabric of the firstdevice.
 22. The method of claim 21 wherein the source identifiercorresponding to a phantom representation in the second fabric of thefirst device is determined from information contained in the globalheader.
 23. A method for routing Fibre Channel packets between a firstdevice in a first fabric and a second device in a second fabric, themethod comprising: receiving a packet in the second fabric, the packetcomprising: a frame header including a source identifier correspondingto the first device in the first fabric and a destination identifiercorresponding to the second device in the second fabric; and a globalheader comprising a global source identifier and a global destinationidentifier; changing the source identifier to correspond to a phantomrepresentation in the second fabric of the first device; and passing thepacket to the second device.
 24. The method of claim 23 wherein thesource identifier corresponding to a phantom representation in thesecond fabric of the first device is determined from informationcontained in the global header.
 25. The method of claim 23 furthercomprising removing the global header from the packet before passing thepacket to the second device.
 26. A switching device for routing FibreChannel packets between a first device in a first fabric and a seconddevice in a second, interconnected fabric, the switching devicecomprising an ingress module connected to the first fabric configured toreceive a Fibre Channel packet from the first device and change adestination identifier of the packet from a phantom representation ofthe second device in the first fabric to correspond to the second devicein the second fabric.
 27. The switching device of claim 26 furthercomprising an egress module configured to change a source identifier ofa packet received from the second device in the second fabric tocorrespond to a phantom representation in the first fabric of the seconddevice
 28. The switching device of claim 27 wherein the ingress moduleand the egress module are the same module.
 29. A switching device forrouting Fibre Channel packets received from a source device in a remotefabric to destination device in a local fabric, the switching devicecomprising an egress module configured to change a source identifier ofthe packet to correspond to a phantom representation in the local fabricof the source device.
 30. A switching device for routing Fibre Channelpackets between separate interconnected fabrics, wherein the switchingdevice is configured to implement a method according to claim
 18. 31. Aswitching device for routing Fibre Channel packets between separateinterconnected fabrics, wherein the switching device is configured toimplement a method according to claim
 21. 32. A switching device forrouting Fibre Channel packets between separate interconnected fabrics,wherein the switching device is configured to implement a methodaccording to claim
 23. 33. A network comprising one or more switchingdevices according to claim
 26. 34. A network comprising one or moreswitching devices according to claim 29.